Deciding when to have sex for the first time

I have recently started university and moved away from home, and also I have met a guy who is now my boyfriend; it is going to be three weeks since we started dating and it has all been really good. We also have already started talking about making love and my boyfriend isn't forcing me into anything, and when I'm with him, it feels right, but the only thing that is bothering me is the fact it will only be three weeks. Both of us are virgins and I know he is sure he wants to lose his virginity to me and I feel the same way, but the time factor really bothers me. I really want to make this relationship work, and I think I'm going to be in love with him very soon...